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AUTOMATIC FILM GRAIN MODELING IN THE FREQUENCY DOMAIN 

p^^rtRnilND OF THE INVENTION 

Film grain simulation has been implemented in several commercially available 
5 products. It is often used to blend a computer-generated object into natural scenes. 
Cineon (trademark of Eastman Kodak Co.) was apparently the first digital film 
application to implement grain simulation. Cineon produces very realistic results for 
many grain types but requires manually setting the film grain parameters. 

Another commercial product allowing the simulation of film grain is Grain 
10 Surgery from Visual Infinity Inc., which is used as a plug-in of Adobe ® After Effects 
®. Their approach seems similar to that described in US Patent No. 5,629,769 
issued to R.E. Cookingham and P.J. Kane (hereinafter Cookingham et al.) entitled, 
"Apparatus and method for the measurement of grain in images". Cookingham et al. 
suggests generating synthetic grain by filtering a set of random numbers. This 
15 product provides algorithms allowing film grain generation with user interaction, but 
also allowing non-aided film grain simulation provided a grainy source as reference. 

Motion picture film typically contains signal-dependent noise resulting from the 
physical process of exposure and development of the photographic film, which 
originates a characteristic quasi-random pattern, or texture, caused by the physical 
20 granularity of the photographic emulsion. Alternatively, signal-dependent noise can 
also be the result of subsequent editing of the Images. We will refer to this signal- 
dependent noise as grain or film grain. 

A similar pattern might be simulated for video compression purposes. In 
previous disclosures, we proposed a method for encoding film grain as supplemental 
25 enhancement Information for a coded sequence. More specifically, we proposed to 
filter grain out of images before compression, then transmit compressed video 
together with a message containing information about original grain, and let the 
decoder restore original grainy appearance of images by simulating film grain based 

on the content of the message. 
30 To the best of our knowledge, no prior art ref erences exists describing non- 

aided methods that characterize the grain of a reference source. 
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qi 1MMAKY OF THE IN VENTION 

In accordance with the principles of the present invention, we disclose a 
method that allows modeling the film grain in the frequency domain. Such method 
5 can be used to automatically parameterize film grain, or to initialize a user-aided 

process of film grain modeling. 

A specific application of this invention concerns the representation of film 
grain, for which the method estimates: (1) the noise variance that models the 
perceived intensity of the grain; and (2) the cut frequencies that characterize the size 
10 of the grain when filtering in the frequency domain. The film grain parameters 

estimated by this method could be conveyed in accordance with the H.264 1 MPEG-4 
AVC standard in an SEI message allowing film grain reinsertion at the decoder end. 



nPTAll FP> DFRCRIPTION 
15 In accordance with the principles of the present invention, we disclose a 

method for allowing automatic modeling of signal-dependent noise in the frequency 
domain. A specific application of this invention concerns the representation of film 
grain In accordance with models previously disclosed in prior patent applications. 
Formulation of this model is as follows: 



20 



25 



30 



Igralnt X, y, C ] = Iwlthout grain [ X, y, C ] + G[ X, y, C] 



0) 



where G[ x, y, c] represents the simulated grain at pixel coordinates (x, y) and 
color component c. In the frequency domain, G[ x, y. c ] is computed as 



G[x,y,cl= p*Q[x,y.c] + u*Glx,y,c-1 ]. 



(2) 



where parameter p is the standard deviation of the random noise and 
parameter u models the cross-color correlation among different color components. 
More in detail, Q[ c ] is a two-dimensional random field generated by filtering blocks 
b of NxM random values, which have been generated with a normalized Gaussian 
distribution N(0,1). The band-pass filtering of blticks b N can be performed in the 
frequency domain as follows: 



2 



PU030259 



Step 1 : Transform 

B = DCT_NxM( b) 

5 

Step 2: Frequency filtering 

for(y=0;y<N;y++) 
for( x= O; x<M; x++) 
10 if ( (x<LOW_HF && y<LOW_VF) II 

x>HIGH_HF II y>HICaH_VF) 
B[x,y] = 0; 



15 Step 3: Inverse transform 
b* m IDCT_NxM( B) 



20 



25 



30 



Finally, Q[ o 3 is formed by mosaicing the filtered blocks b'. To reduce possible 
blockiness, the algorithm may introduce a last step in which a low-pass filter appl.es 

to block transitions. 

Although M and N could take any value, most common implementafions 
would likely use squared blocks of 16x16, 8x8 or 4x4 pixels. Note also that other 
transforms, such as the Fast Fourier Transform (FFT), could be used in steps 1 and 
3. instead of the Discrete Cosine Transform (DCT) used in the equations. 

,n accordance to this model and in accordance with the principles of the 
present invention, one embodiment of the instant invention estimates: 

The noise variance that models the perceived intensity of the grain (parameter 
p in equation (2); and 
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The horizontal and vertical cut frequencies (LOW_HF, HIGH_HF, LOW_VF, 
HIGH__VF) that characterize the size of the grain when filtering in the frequency 
domain. 

5 Assuming there is no cross-color correlation (u=0 in equation (2)). the f ilm 

grain parameters estimated by this method could be conveyed in accordance with 
the H.264 1 MPEG-4 AVC standard in an SEI message allowing film grain reinsertion 
at the decoder end. 



The following sections present the different processes used in accordance 
with the principles of the present invention: 

1. Rim grain extraction 

15 In all cases, film grain parameterization requires as a first step to extract the 

film grain from the original source. When two versions of the original content are 
available, one with film grain and another without, a perfect extraction of the film 
grain is possible by subtraction, as illustrated in Figure 1 . When the version without 
film grain is not available, a filtered version of the original source can be used as an 
20 estimate of the image without film grain, as illustrated in Figure 2. 

In the latter case, however, the residual obtained by subtracting filtered from 
original images will provide film grain mixed with other fine details and textures 
removed by the filtering process. To avoid introducing a bias in the film grain 
modeling process, in one embodiment of the present invention, we propose an 
25 alternative extraction procedure, illustrated in Figure 3. 

Through this procedure, the flat zones of the filtered images are detected prior 
to film grain extraction. Flat zones can be detected on a pixel basis or on a block 
basis with different criteria of flatness. In a particular embodiment, the algorithm 
would signal as flat zone any block of NxM pixels presenting a variance lower than a 
30 given threshold. Based on the flat zone detection, the algorithm outputs a mask 
differentiating between flat zones and non-flat zones. The extraction algorithm will 
use this mask to restrict the film grain extraction to the flat zones. Several images 
can be filtered and used as source of flat zones/blocks. To avoid running out of flat 
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zones/blocks from where to extract grain, the algorithm could pick the M flattest 
zones/blocks, for a required M. 

The system of Figure 3 requires more memory bandwidth than the systems of 
Figures 1 and 2, because for each original image, a mask image signaling the flat 
5 zones is stored and accessed. However, it provides improved performance since it 
isolates the film grain from textured regions where other high frequencies might have 
been suppressed by the filtering process. 

2. Film grain characterization 

10 Following the film grain extraction, the system proceeds to the film grain 

characterization. In accordance with one embodiment of the present invention, film 
grain characterization in the frequency domain is comprised of two blocks, as 
illustrated in Figure 4. 

15 2. 1 . Estimation of the noise statistics 

Some characteristics of the f ilm grain can be well represented by the 
statistical values of the grain image. One step of film grain characterization is 
computing the statistics of the noise, represented by Block 1 in Figure 4. These 
values may comprise both low- and high-order statistics such as mean, variance, 

20 skewness or kurtosis. In the particular implementation of grain simulation presented 
in [3.4], only the standard deviation is used to determine the intensity of the grain 
assuming zero mean. Following the notation introduced in equation (2), parameter p 
is estimated as: 

25 P=O g rab> 

Although several images may be present at the input of the noise statistics 
computation block, it is not mandatory that all their pixels to be used in the 
estimation. In a particular embodiment, the statistics may be computed by 
combining estimates on different regions of the grain images to facilitate hardware 
implementations (for example, parallelization). 



30 



5 



PU030259 



2. 2. Estimation of the spatial correlation in th fr qu ncy domain 

Block 2 of Figure 4 is the estimation of the spatial correlation in the grain 
images, which will convey a representation of the shape and size of the film grain. In 
5 accordance with the principles of the present invention, we describe a non-aided 
method that estimates the cut frequencies characterizing the spatial correlation of 
the film grain in the frequency domain. These frequencies may be used as 
parameters of the earlier-mentioned SEl message which would allow film grain 
reinsertion at the decoder end. 
10 A particular implementation based on the DCT of squared blocks of NxN 

pixels is presented in Figure 5. In the first step, a block of NxN pixels is extracted 
from the input grain images previously obtained. Then, the DCT of the block is 
computed and the resulting coefficients are stored in the block stores. Finally, a 
condition is assessed to decide if more blocks are needed. In the trivial 
15 implementation, all input blocks are stored in the block stores, but other approaches 
could be used in order to reduce memory requirements or computational load. To 
cite a few, a possible implementation would stop processing once a certain number 
of blocks was reached, while another implementation could rely on the number of 
blocks extracted from each grain image In order to decide when stop. 
20 Once the block stores is full, a mean block B me an is computed by averaging 

frequency coefficients from all blocks in the block stores. Assuming K as the number 
of blocks in the block stores, the averaging process for the frequency coefficient at 
position [x,y] can be formulated as follows, 

1 (4^ 

25 B mOT ,[*.:y]=^XB,[*, y ] V ; 

In the next step, the algorithm computes the horizontal and vertical frequency 
vectors. In the block diagram of Figure 5, both processes are computed in parallel. 
We have defined the horizontal frequency vector B H as the vector resulting 
30 from averaging the N frequency coefficients of each row of B mean 

l £=J (5) 
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Similarly, have defined the vertical frequency vector B v as the vector resulting 
from averaging the N frequency coefficients of each column of B mea n 



15 



4 l £=? 

b v m=— 2-B^rx.n] 



(6) 



Based on the frequency vectors, the last step in the estimation of the spatial 
correlation of the film grain is detecting the cut frequencies in B„ and By. if any. In 
the block diagram of Figure 5, these calculations are done in parallel. The procedure 
10 that extracts the cut frequencies for B H in one embodiment of the present invention is 
disclosed below; however, it should be noted that the same procedure may also be 
used for By. 



The estimation of the cut frequencies is done as follows: 

Step 1: low-pass filter the components of the frequency vector to avoid spurious 
peaks, in a particular implementation, this can be done by convolving the 

frequency vector with a filter of impulse response h[n]: 



20 B' H [n] = l,B H ti]htn-n = (B l , *fc)[n] 



(7) 



In the filtering process, it is desirable to avoid the influence of the DC term 
located at position 0 in the frequency vector. For example, a 3-tap linear filter 
25 with coefficients w 0 , w 1f and w 2 should apply only to the coefficients with index 

n^2: 



B'„W = w,-B f( [n-l]+«' 1 B B [»]+w 1 .B ( ,[ B+ ll, 2<»<iV-l 



30 

Step 2: compute the mean value of B' H by averaging its components: 



(8) 
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- 1 fc 1 (9) 



5 Step 3: representing B' H as a curve, compute its intersection points with the average 

value B' H , then: 

If a single intersection point is found, the index n of the closest 

component in B' H is chosen as the value of the horizontal high 
cut frequency; the horizontal low cut frequency is assumed to be 

10 0. 

If two intersection points are found, the indexes of the closest 

components are found for each one. The lowest value will 
correspond to the low horizontal cut frequency whereas the 
highest value will correspond to the high horizontal cut 

15 frequency. 

If more than two intersection points are found, no spatial correlation is 
detected. The horizontal low cut frequency is assumed to be 0, 
and the horizontal high cut frequency is assumed to be N-1 , 
indicating to the film grain simulation function that no frequency 

20 filtering is required to imitate the original grain. 

The same procedure may be applied for the vertical frequency vector B v . At 
the end of both processes, the algorithm yields four cut frequencies (LOW_HF, 
HIGH.HF LOW_VF, HIGH_VF) that allow characterizing both the size and the 
25 elongation of the grain. Elongated grain will result when LOW.HF # LOW_VF and/or 

HlGH_HF * HIGH.VF. 

As an alternative approach, it is possible to constrain the grain to circular 
shapes. This implies that horizontal and vertical cut frequencies will be the same. In 
this case, vertical and horizontal frequency vectors (B H and Bv) are averaged to 
30 create single frequency vector (B). Then, the same procedure is used to est«mate 
low and high cut frequencies. 
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Figure 1. FOro grain extraction. 
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Figure 2. Film grain extraction. 
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Figure 3. FUm grain extracts 



\ 

\ 



PU030259 



Sheet k Of 5 



Film grain characterization 



Grain images 



Block 1 



Estimation of the 
noise statistics 



variance 



Block 2 



Estimation of the spatial 
correlation in the frequency 
domain 



cut 
frequencies 



-> parameters 



figure 4. Film grain characterization. 
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Figure 5. Estimation of the spatial correlation in the frequency domain. 
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